Automatic Generation of Versatile Benchmarks forParallel Production System Architectures

The shortage of adequate benchmarking facilities is a major problem in the proper evaluation of production system machine organizations. This paper presents a new benchmark problem that allows independent variations in the size of the database, the number of productions , the ratio between local and global data, and the variance in the size of local data clusters. This benchmark, available via the Internet through anonymous ftp, is based on the traditional Traveling Salesperson Problem and is simple yet versatile. It can be used to evaluate parallel production system machines, including those for which the criteria for correct concurrent execution is based on the serializability principle. The advantages of our benchmark are illustrated through its application in the performance evaluation of a recently proposed concurrent production system architecture. Our experiments indicate that the performance of this architecture scales with the size of the database, but is degraded by a database with high variance in the size of local data clusters.